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Despite of much focus on professional development aimed specifically at developing teachers’ technology 
integration skills, rigorous studies of effective PD (professional development) are lacking. Evidence is also lacking 
on how these skills can best be integrated with pedagogical and content knowledge to improve student learning. 
The purpose of this article is to present two “design-oriented” TTPD (technology-related teacher professional 
development) designs and investigate the designs’ impact on teachers. In one TTPD ( tech-only ), teachers learned 
technology skills to create activities using online learning resources. In the other ( tech+PBL ), teachers learned to 
create PBL (problem-based learning) activities using online resources. All teachers implemented these activities 
with their students. Findings indicate similarities and differences across several outcomes, including teacher 
knowledge, teacher attitude, usage of PBL and Web usage data. In addition, an instrument was developed to 
measure the students’ self-reported knowledge, attitudes and behavior regarding their teachers’ implemented 
activities. The instrument was shown to be valid and reliable. 

Keywords: technology-related teacher PD (professional development), PBL (problem-based learning), math and 
science education 


Introduction 

The increased pervasiveness of Internet technologies in school settings provides an instant access to a 
growing Network of high quality and open access “online resources” for education (Ainsworth, Honey, & 
Johnson, 2005; McArthur & Zia, 2008; Borgman et al., 2008). These online learning resources include a wide 
array of simulations, data sets and lesson plans. As such, they have a substantial, yet largely untapped potential 
to support teachers in creating tailored activities that enhance diverse students’ educational experiences. 
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However, as widely documented, teachers often lack the time and technology skills necessary for effective 
technology integration (Hanson & Carlson, 2005; Kramer, Walker, & Brill, 2007). As such, teachers need 
support in developing their capacity to teach effectively in 21st century classrooms. 

Studies have shown that teacher PD (professional development) is an effective way to improve teacher 
skills, knowledge and attitudes (Borko, 2004). The increase in technology use in school has seen a concomitant 
increase in technology-related professional development (Means, Murphy, Javitz, Haertel, & Toyama, 2004). 
Within this arena of research, efforts have focused on conceptualizing and measuring the intersection of 
TPACK (teacher technological, pedagogical, and content knowledge) and corresponding ways to improve that 
knowledge set through professional development (Angeli & Valanides, 2005; Koehler & Mishra, 2005, 2008). 
Alongside the rich literature base, there are several disagreements about fundamental tenets of TPACK and 
much work remains to improve the theoretical base for the construct (Angeli & Valanides, 2009). At times, the 
practice of measuring TPACK appears to be at odds with its theoretical definition. For example, there is a tacit 
assumption among some that TPACK is a constitutively defined construct. 

Yet, despite much PD aimed specifically at developing technology integration skills, rigorous empirical 
studies of effective PD is lacking. Moreover, evidence is lacking on how newly learned technological skills can 
best be integrated with pedagogical and content knowledge in ways that improve student learning (Lawless & 
Pellegrino, 2007) 

The purposes of this article is to present two design-oriented technology integration professional 
development (TTPD) models and investigate their impact on teacher and student learning. In one design 
(i tech-only ), teachers learned technology skills coupled with a self-chosen pedagogy to create student activities 
using online resources. In the other ( tech+PBL ), teachers learned technology skills to create inquiry-oriented 
(specifically PBL (problem-based learning)) activities for their students using online resources. In this way, our 
study compared the impact of a TTPD design focused on integrating technology with a self-chosen pedagogy 
alone, with one integrating technology and pedagogy. 

The Theoretical Framework 

Teacher professional development has long been a way to increase teachers’ skills and studies have 
demonstrated its positive effects on instructional practices and resultant student learning (Borko, 2004). Previous 
studies have identified general characteristics of effective PD (e.g., intensive, sustained, job-embedded, 
collaborative, active and content focused). However, rigorous evidence of effective PD, especially with regards to 
long-term impacts on teacher and student learning is lacking (Wayne, Yoon, Zhu, Cronen, & Garet, 2008). 

The TTPD design used in the present research is design-oriented in that participants learn to design 
instructional activities for their students. Proponents of design-oriented PD argue that this approach enables 
teachers to learn new technology skills within an authentic instructional context. This helps them take 
ownership of new skills, making them more likely to integrate these into future teaching (Lawless & Pellegrino, 
2007). This perspective also fits with a more contemporary view of teaching as a kind of design task, in which 
teacher adaptation and use of materials are seen as a critical step in curriculum design (Brown & Edelsen, 2003; 
Remillard, 2005). 

The Technology Context 

The technological context for the TTPD is a free and Web-based tool, called the IA.usu.edu (instructional 
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architect). The IA supports teachers as designers of instructional content, allows them to create instructional 
activities for their students using online resources and package their designs as Web pages. There are several 
different ways for teachers to use the IA. Within “My Resources” area, teachers can search for online resources 
from the Web, and once found, save them into folders. Resources can include anything with a valid URL 
(Uniform Resource Locator), such as RSS (Really Simple Syndication) feeds, podcasts, Web pages, PDF 
(Portable Document Format) documents or graphics among others. The “My Projects” area (see Figure 1) 
allows teachers to select from their saved online resources, group them together, sequence them, and provide 
directions, lead questions, prompt or other form of annotation to create learning activities (called IA projects 
throughout the remainder of the manuscript article). The “Publish” area gives teachers control over who can view 
their work, which can be completely private, viewable only by students with a login or completely public to 
anyone with Web access. In addition, teachers who are logged in have the ability to copy public projects from 
other teachers. This allows them to take an existing project and adapt it for their own classroom use. 



Figure 1. Illustration of a user managing IA projects in the “My Projects” area. 



You will follow the directions of your teacher as you move through these different maps. 

Here is a link to the explanation of the project for teachers: Madagascar explanation 
You should have two copies of this map to complete the exercise: Blank madaaascar map 

Problem Presentation 

Situation: You have found yourself moving to a 

large island and need to determine where you want to build a community. Now 
look at the information on this map. What do you see that might influence where 
people live on the island Which quadrant do you think they live in? Utilizing 
all of the information available here about the weather systems, land coverage, 
water features, and terrain, and through all the information you can find, you 
need to determine where would be the best place to build. You will need to 
justify your response and provide evidences of what would make that location a 
beneficial local to live based on the research you perform. 

Figure 2. Example IA project, showing teacher annotations (text) and two forms of online resources: Hyperlinks and 



an embedded graphic. 
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Since 2005, the IA has over 7,100 registered users who have gathered over 70,000 online resources and 
created over 15,900 IA projects. Since August, 2006, public projects have been viewed over 1.5 million times. 
Figure 2 shows an example IA screen shot with a teacher-created inquiry activity and an embedded online 
resource. 


Research Design 

The study was conducted to compare the impact of two TTPD designs, using a non-equivalent pre-/post- 
test control group design (Campbell & Stanley, 1963). Table 1 shows the study’s research questions, data 
sources, and analyses. 

Table 1 


Research Questions , Data Sources and Analyses 


Research question 

Data source 

Analysis 

1. What is the impact of the two TTPD designs on teachers’ 
knowledge about, attitude towards, and experience with 
technology integration? 

Teacher pre-/post- survey 

Descriptives 
Friedman tests 

2. What is the impact of the two TTPD designs on teachers’ 
usage of the IA? 

Web usage data 

Descriptives 

3. What is the impact of the two TTPD designs on PBL use in 
designing IA projects? 

PBL alignment score 

Descriptives 
Mann- Whitney U tests 

4. Is the self-reported student questionnaire a valid and reliable 
instrument? 

Student questionnaire 

Descriptives 
Factor analysis 


PD (Professional Development) Designs 

The TTPD designs were a series of three workshops with in-between activities and were conducted 
face-to-face over three months. Following key principles of effective PD, they were sustained, content focused, 
active, and collaborative (Borko, 2004). Following design-oriented approaches in technology integration 
professional development (Lawless & Pellegrino, 2007), the participants engaged with authentic and complex 
problems in their own teaching. In an effort to resolve those problems, teachers designed solutions, 
implemented their designs in the classroom and reflected with their peers on the instructional efficacy of their 
designs. 

In terms of technology content, the TTPD designs focused on the following technology skills: (1) finding 
and using online resources; (2) designing activities for students using the IA; and (3) implementing these IA 
projects in the classroom. In one design ( tech-only ), participants solely learned skills to design activities for 
their students using the IA, a self-chosen pedagogy, and online resources and the IA. In the other design 
( tech+PBL ), participants learned technology skills to design inquiry-oriented activities for their students using 
online resources and the IA. The particular inquiry approach was PBL, wherein students acquire knowledge 
through engaging with authentic problems (Barrows, 1986; Barrows & Tamblyn, 1980; Savery, 2006). In PBL, 
learners operate in small groups to solve authentic problems using resources made available to them. The 
instructor facilitates, scaffolds, coaches, and designs problem-solving behaviors (Hmelo-Silver & Barrows, 
2008). Each problem cycle concludes with a reflection phase. PBL was selected as the TTPD approach with 
teachers in part, because prior research has proven effective both for teacher education ( d = 0.64), and when 
participants are engaged in designing problems {d = 0.74) (Walker & Leary, 2009). 

Table 2 shows key activities for the two TTPD designs, as well as data collection points. Note that 
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participants in the tech+PBL TTPD design were asked to utilize PBL with their stud ents, only if they felt it 
aligned with their self-selected design problem, student needs, and their own educational philosophy. In 
contrast, participants in the tech-only TTPD design were asked to design activities using the IA in ways that 
aligned with their own educational philosophy. 

Table 2 


Key Activities for the Two TTPD Designs and Data Collection Points 


Tech-only TTPD 

Tech+PBL TTPD 

Data collected 

Workshop 1 (3 hours) 

(1 ) Take pre-survey 

(2) View example IA projects 

(3) Intro to online resources 

(4) Intro to IA: Walk through sample project 
creation 

(5) Participants select design problem 

(6) Individuals design IA project(s) 

(7) Discuss selection of quality of online 
resources 

(8) Review IA functionality 

Same 

Teacher pre-survey 

Classroom implementation #1 

(1) Design and implement IA project(s) with 
students 

(2) Provide reflection paper on barriers and 
successes in implementation 

Same 

(1) Student pre-/post- 
questionnaire 

(2) PBL alignment of IA 
project 

(3) Web usage 

Workshop 2 (3 hours) 

(1) Small then large group discuss 
implementation experiences 

(2) Review use of the IA. including advanced 
features 

(3) Design a new I A learning activity 

(4) Share ideas 

(5) Individuals begin to design new I A 
project(s) 

(1) Small then large group discuss implementation 
experiences 

(2) Review use of the I A 

(3) Engage in inquiry-oriented activity 

(4) Large group inquiry-oriented discussion 

(5) Design own PBL learning activity 

(6) Share ideas 

(7) Individuals begin to design new IA project(s) 

- 

Classroom implementation #2 

(1) Design and implement new IA project(s) 
with students 

(2) Write reflection paper on barriers and 
successes 

Same 

(1) Student pre-/post- 
questionnaire 

(2) PBL alignment to IA 
project 

(3) Web usage 

Workshop 3 (3 hours) 

(1) Small then large group discuss 
experiences: Technology 

(2) Review technical use of the IA, including 
advanced features 

( 1 ) Small then large group discuss experiences: 
Technology 

(2) Review technical use of the IA 

(3) Small then large group discuss PBL implementation 
experiences 

Teacher post-survey 


Methods 


Participants 

Classroom teachers from two adjacent school districts ( N = 18) were assigned (based on scheduling 
preference) to one of two TTPD designs. In one design ( tech-only ), teachers ( N = 9) solely learned technology 
skills to design activities for their students using online resources. In the other (tech+PBL), teachers ( N = 9) 
learned technology skills to design specifical PBL activities for their students using online resources. Teachers 
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were allowed to select one preferred classes in which to implement their design activities. Table 3 and Table 4 
show the teacher and student demographics, respectively. 

Teachers received one university course credit for completing all requirements. 

Table 3 


Teacher Demographics 


Teacher demographic 

Tech-only TTPD 

Tech+PBL TTPD 

# of teachers (% female) 

9 (100%) 


9 (88.9%) 

Mean (standard deviation) of years in the 



current position 

10.33 (6.22) 


12.33 (9.90) 

Table 4 




Student Demographics 




Characteristic 

Tech-only N (%) 

Tech+PBL N (%) 

TTPD design 


226 (67.87) 

107 (32.13) 


White 

164 (72.57) 

84 (78.50) 


Hispanic/Latino 

29(12.83) 

6 (5.61) 


Black or African American 

1 (0.44) 

0 

Ethnicity 

Other- 

6 (2.64) 

1 (0.93) 


Two or more groups 

7(3.10) 

7 (6.54) 


Did not answer 

19 (8.40) 

9 (8.41) 


English 

193 (86.02) 

101 (94.39) 


Spanish 

19 (8.41) 

5 (4.67) 

Primary language 

Bilingual 

1 1 (4.87) 

1 (0.93) 


Other 

3(1.32) 

0 


Data Sources 

Table 2 shows TTPD activities and data collected at each phase. 

Teacher survey. Teacher data were collected using an online survey at the beginning and end of the 
TTPD. The survey consisted of five Lickert scale (0 = “Strongly disagree”; 4 = “Strongly agree”) and eight 
open-ended items. Items were adapted from an established measure (Becker, 2000) of teacher knowledge, 
attitudes, and experience with respect to technology and teaching. 

Table 5 


Teacher Survey Sub-scales 


Sub-scale 

# of Lickert-scale item 

Max. total points possible 
(0 = Low; 4 = High/item) 

Reliability 

Knowledge 

2 

8 

0.89 

Attitude 

2 

8 

0.66 

Experience 

2 

8 

0.26 


Responses on items for each sub-scale were summed. All teachers completed pre- and post- survey and 
r-tcsts of pre-test results showed no significant differences between groups (p > 0.05) for all of the subscales. In 
terms of reliability, teacher’s knowledge and attitude sub-scales showed high reliability while the reliability for 
the experience sub-scale was very low (see Table 5). Therefore, teacher experience was excluded from the 
further analysis and discussion. 

Web usage data. Data of teachers’ use of the IA (Khoo, Pagano, Washington, Recker, Palmer, & 
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Donahue, 2008) were automatically collected by the IA system, including the number of IA logins, IA projects 
created, collected online resources used and student visits to IA projects. 

PEL alignment of IA projects. In order to score alignment of IA projects with PBL, we developed a PBL 
rubric using items based on Walker and Shelton (2008). The rubric consisted of 14 elements in four categories 
(see Appendix). Three raters, randomly selected from a pool of five and blind to TTPD condition, 
independently scored the PBL alignment of teachers’ IA projects. Each element’s score ranged from 0 to 1 (0 = 
“Not present”; 1 = “Present”), for a maximum possible score of 14 points. The overall average one-way random 
effects ICC (intra-class correlation) (Shrout & Fleiss, 1979) was 0.83, which suggested an almost perfect 
agreement (Sim & Wright, 2005). 

Student questionnaires. Students completed questionnaires before and after each of the two IA project 
implementations. As teachers taught different courses, an achievement test of student knowledge was not 
feasible. Instead, the student questionnaire contained self-report Likert-scale items addressing student 
knowledge, attitude and behavior. 

Results 

Results are organized by research questions. A variety of statistical testing was done, with a uniform alpha 
level of 0.05 for each test. 

Research Question #1: Impact on Teachers 

Impacts on teacher knowledge and attitudes in technology integration were assessed using the pre-/post- 
survey. Descriptive statistics and effect sizes (Cohen’s d) of within-group gains are shown in Table 6. 

Due to the small sample size, a Friedman analysis of variance test was conducted to compare the pre- and 
post- survey scores. Overall, teachers showed significant gains in knowledge, Friedman = 10.89, p < 0.001. 
There was no significant gain in teacher attitudes (p = 0.48). In part, this may be because participants reported 
higher attitudes on the pre-survey, and thus, had little room for improvement. 

In terms of the group difference, a Friedman analysis of variance test was conducted to compare the gain 
scores between the TTPD designs. The result showed no group difference for either knowledge gain (p = 0.51) 
or attitude gain (p = 0.51), which may be a result of the relatively low statistical power and small n involved. 
When examining pre -/post- changes in terms of effect size, both TTPD design showed large gains for teacher 
knowledge. However, for teacher attitudes, the tech-only group showed a negative value, while the tech+PBL 
group showed a medium effect size. 

Table 6 


Teacher Self-report on Technology Integration Knowledge and Attitudes 




Pre- 

■survey 

Post- 

■survey 

Cohen s d 



Mean 

SD 

Mean 

SD 


Tech-only (N = 9) 

Knowledge using technology 
in classroom 

5 

1.87 

7.22 

1.20 

1.45 


Attitude in teaching with technology 

7.44 

0.88 

6.88 

2.67 

-0.32 

Tech+PBL (N=9) 

Knowledge using technology 
in classroom 

4.56 

2.30 

6.56 

0.88 

1.26 


Attitude in teaching with technology 

6.44 

1.33 

7 

1.12 

0.46 


Note. Possible values range from 0 = Low to 8 = High. 
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Research Question #2: Impact on IA Usage 

Table 7 shows summary IA usage statistics for the two TTPD designs. Overall, usage is high, with high 
numbers of participant logins, IA projects created, and collected online resources used. Student usage also 
appears high, with high numbers of visits to the IA projects created by teachers, including one IA project 
accessed over 500 times. In short, these data suggest that the IA was successfully used by both teachers and 
students. 


Table 7 

The IA Usage Data (Six Months After the Completion of the TTPD ) 




Mean 

SD 

Max. 


# of participant logins to the IA 

37.78 

24.85 

95 

Tech-only TTPD (N = 9) 

# of IA projects created 

9.22 

3.23 

14 

# of collected online resources used 

36.67 

31.50 

114 


# of visits to non-private IA projects 

150.44 

137.77 

423 


# of participant logins to the IA 

45.56 

19.77 

69 

Tech+PBL TTPD (N = 9) 

# of IA projects created 

14.67 

6.60 

26 

# of collected online resources used 

70.22 

37.72 

141 


# of visits to IA projects 

109.89 

168.81 

545 


Research Question #3: Impact on Design 

Table 8 shows results for teachers’ IA projects PBL alignment scores. Note that PBL alignment results are 
likely an under-estimate of what happened in the classroom. Teachers may have asked students to use the IA 
project in groups, as an example, even though the IA project content did not make that clear. The means for all 
PBL scores are quite low, which may be the result of this underestimation, an overly strict measure, or may 
suggest that the PBL portion of the TTPD was not effective. 

Table 8 


IA Project PBL Alignment Scores 




Mean 

SD 

Max. 

Tech-only TTPD (N = 9) 

PBL score after workshop 1 

0.22 

0.55 

0.67 

2 

PBL score after workshop 2 

1.33 

4 

Tech+PBL TTPD (N = 9) 

PBL score after workshop 1 

0.33 

0.71 

0.67 

2 

PBL score after workshop 2 

0.22 

2 


Note. Possible values range from 0 = Low to 14 = High. 


Although scores were based on scales, a Mann-Whitney U test was run to account for the small sample 
sizes. Comparisons between different workshop treatments were not statistically different both after the first 
workshop (p = 0.58) and the second workshop (p = 0.54). In short, the TTPD did not change teachers’ usage of 
PBL over time and both groups used consistently negligible levels of PBL in their IA projects. As noted, this 
may be a result of an overly stringent rubric. 

Research Question #4: Validity and Reliability of Students Questionnaires 

The ultimate test of the effectiveness of professional development is determining its links with student 
learning, although these links are likely to be indirect (Fishman, Marx, Best, & Tal, 2003). Our approach was to 
provide pre-/post- questionnaires to participants’ students at the start and end of an activity using an IA project. 
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However, it is most necessary to determine if the measurement was valid and reliable. 


Table 9 

Student Survey Items Factor Analysis 


Factor 
Factor 1: 

Student knowledge 

Factor 2: 

Student attitude 


Factor 3: 

Student behavior 


Survey item 

I know enough to teach my friends about this topic 
I know a lot about this topic 

If I got to decide what to do in class next I would pick this topic 
I like the topic the teacher has selected very much 
I spend time outside of school learning about this topic 
I talk with my friends about this topic 
I talk to my parents about this topic 


Factor loading 
0.88 

0.88 

0.89 

0.88 

0.79 

0.76 

0.76 


For the purposes of validity, a confirmatory factor analysis with varimax rotation showed three total 
factors (see Table 9). All were precisely aligned to the sub-scales as planned. Factor loading ranged from 0.76 
to 0.89. Given the combination of a large sample size ( N = 333) and the strength of factor loadings (Stevens, 
1999), these data appear to be valid measures of student self-reports for behavior, knowledge, and attitude. 

Table 10 provides details regarding the number of items whose responses were summed on the teacher 
survey for each of the constructs measured. Overall, questionnaire reliability was high ( a = 0.79), as was each 
of the sub-scales. 


Table 10 


Pre-/Post- Student Questionnaire of Behavior, Knowledge, and Attitude Sub-scales 


Sub-scale 

# of Lickert-scale item 

Max. total points possible 

(1= “Strongly disagree”; 5= “Strongly 

agree’Vitem) 

Reliability 

Student knowledge 

2 

10 

0.80 

Student attitude 

2 

10 

0.71 

Student behavior 

3 

15 

0.73 


Discussion and Conclusions 


This article reported the impact of two design-oriented TTPD. Both showed high teacher and student 
usage of the tools and significant gains in teacher reported knowledge. Although both failed to improve teacher 
attitudes, this may be in part due to teachers’ initially high levels of attitude leaving little room for 
improvement. When comparing the gain scores between TTPD groups, none of the TTPD design statistically 
outperformed the other on either the knowledge or attitude measures. However, in terms of the effect sizes, 
knowledge gain was large for both groups. For attitudes, the tech+PBL group showed a medium gain, while the 
tech-only group showed a negative gain. 

Analyses of teachers’ IA projects showed overall low usage of PBL elements, perhaps partially due to the 
fact that the use of PBL was optional. Furthermore, while the high inter-rater reliability evidence for the PBL 
alignment rubric is encouraging, the low scores of use of PBL elements show that measurement work remains. 
It is possible that PBL alignment scoring was not sensitive enough to differences within teacher-designed 
activities. For example, IA projects not only are needed to be cross-disciplinary, but also present 
cross-disciplinary problems to students, a rather high bar. As such, future work should consider refining the 
PBL rubric. 
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Finally, from a measurement perspective, the student self-report questionnaires proved to be both valid 
and reliable, as assessed by factor and reliability analyses for this sample. The measure is particularly 
noteworthy, because it was designed for and used with students learning a wide range of subject matter. 

Limitations of the work include the non-experiment research design and the small number of 
participants. Another threat to internal validity may be cross-contamination. Teachers from two TTPD 
designs might have communicated between workshops, shared their experiences, thereby influencing each 
other. In terms of areas for future research, further analysis of the student data should be conducted to link 
teacher practice and student learning outcomes, as well as to compare the impact of different TTPD designs 
on classroom teaching and learning. The student questionnaire presented and evaluated in this article is an 
important step in this direction. 
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Appendix 


Problem-Based Learning Alignment Rating Scale 


PBL element 

Description 

Authentic problems 

Problems are complex (cross-disciplinary). 

Authentic problems 

Problems have multiple solution paths. 

Authentic problems 

Problems are ill-structured. 

Authentic problems 

Problems are likely to be encountered in professional practice. 

Learner centered 
Learner centered 

Learners generate objectives from given (and unresolved) problems. 

Learners are prompted to locate resources (content experts, reference books, journals and 
articles) that will assist in problem resolution. 

Learners are prompted to utilize resources (content experts, reference books, journals and 
articles) that will assist in problem resolution. 

Learners engage in self- and/or peer- assessment of problem solving performance within 
their group. 

Facilitators model and prompt students with meta-cognitive questions that assist in 
problem resolution. 

Learner centered 

Learner centered 

Teachers as facilitators 

Teachers as facilitators 

Facilitators are guides. 

Small group interaction 

Learners interact in groups. 

Small group interaction 

Divide and conquer. 

Small group interaction 

Learners share and discuss their findings. 

Small group interaction 

The group evaluates the utility of the acquired knowledge in solving the problem. 




